Automatic blending of human facial expression and full-body poses for dynamic digital human model creation using integrated photo-video volumetric capture system and mesh-tracking

ABSTRACT

An integrated photo-video volumetric capture system for 3D/4D scanning acquires 3D scans and 4D scans by acquiring images and videos simultaneously. The volumetric capture system for high-quality 4D scanning and mesh-tracking is used to establish topology correspondences across a 4D scanned mesh sequence for generating corrective shapes which will be used in shape interpolation and skeleton driven deformation. The volumetric capture system aids mesh-tracking for maintaining mesh registration (topology consistency) along with ease of extreme pose modeling. Major upper body and lower body joints are able to be identified that are important for generating deformation and capturing the same using a wide range of motion for all movement types across all joint categories. By using the volumetric capture system and mesh tracking, the topology changes are tracked. Each pose captured will have the same topology which makes blending between multiple poses easier and more accurate.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority under 35 U.S.C. § 119(e) of the U.S.Provisional Patent Application Ser. No. 63/169,323, filed Apr. 1, 2021and titled, “AUTOMATIC BLENDING OF HUMAN FACIAL EXPRESSION AND FULL-BODYPOSES FOR DYNAMIC DIGITAL HUMAN MODEL CREATION USING INTEGRATEDPHOTO-VIDEO VOLUMETRIC CAPTURE SYSTEM AND MESH-TRACKING,” which ishereby incorporated by reference in its entirety for all purposes.

FIELD OF THE INVENTION

The present invention relates to three dimensional computer vision andgraphics for the entertainment industry. More specifically, the presentinvention relates to acquiring and processing three dimensional computervision and graphics for film, TV, music and game content creation.

BACKGROUND OF THE INVENTION

Virtual human creation is highly manual, time-consuming and expensive.Rather than hand-crafting Computer Graphics (CG) artwork from scratch,the recent trend is to efficiently create realistic digital human modelby multi-view camera 3D/4D scanners. Various 3D scanner studios(3Lateral, Avatta, TEN24, Pixel Light Effect, Eisko) and 4D scannerstudio (4DViews, Microsoft, 8i, DGene) exist world-wide for cameracaptured based human digitization.

A photo-based 3D scanner studio includes multiple array of highresolution photography cameras. The prior art of 3D scan typically isused to create rigged modeling and requires hand-crafting for animationas it does not capture deformation. A video based 4D scanner(4D=3D+time) studio includes multiple arrays of high frame rate machinevision cameras. It captures natural surface dynamics, but due to fixedvideos and actions, it cannot create novel face expression or bodyaction. Dummy actors need to perform many sequences of actions, meaninga huge workload for the actor.

SUMMARY OF THE INVENTION

An integrated photo-video volumetric capture system for 3D/4D scanningacquires 3D scans and 4D scans by acquiring images and videossimultaneously. The volumetric capture system for high-quality 4Dscanning and mesh-tracking is used to establish topology correspondencesacross a 4D scanned mesh sequence for generating corrective shapes whichwill be used in shape interpolation and skeleton driven deformation. Thevolumetric capture system aids mesh-tracking for maintaining meshregistration (topology consistency) along with ease of extreme posemodeling. Major upper body and lower body joints are able to beidentified that are important for generating deformation and capturingthe same using a wide range of motion for all movement types across alljoint categories. By using the volumetric capture system and meshtracking, the topology changes are tracked. Each pose captured will havethe same topology which makes blending between multiple poses easier andmore accurate.

In one aspect, a method programmed in a non-transitory of a devicecomprises using a volumetric capture system configured for 3D scanningand 4D scanning including capturing photos and video simultaneously,wherein the 3D scanning and 4D scanning includes detecting muscledeformation of an actor and implementing mesh generation based on the 3Dscanning and 4D scanning. The 3D scanning and 4D scanning include: 3Dscans to be used to generate automatic high-fidelity extreme poses and4D scans which include high temporal resolution which enables meshtracking to automatically register extreme pose meshes for blending.Generating automatic high-fidelity extreme poses includes using 3D scansof the actor and muscle deformation of the actor to generate theautomatic high-fidelity extreme poses. 4D scanning and mesh-tracking areused to establish topology correspondences across a 4D scanned meshsequence for generating corrective shapes for shape interpolation andskeleton driven deformation. The method further comprises identifyingand targeting joints and muscles of the actor by the volumetric capturesystem for 3D scanning and 4D scanning Mesh generation includes muscleestimation or projection based on the 3D scanning and 4D scanning andmachine learning. Implementing mesh generation includes using the 3Dscanning and 4D scanning to generate meshes in extreme poses includingmuscle deformation. The method further comprises implementing meshtracking for tracking topology changes to enable each pose captured tohave a same topology for blending between poses.

In another aspect, an apparatus comprises a non-transitory memory forstoring an application, the application for: using a volumetric capturesystem configured for 3D scanning and 4D scanning including capturingphotos and video simultaneously, wherein the 3D scanning and 4D scanningincludes detecting muscle deformation of an actor and implementing meshgeneration based on the 3D scanning and 4D scanning and a processorcoupled to the memory, the processor configured for processing theapplication. The 3D scanning and 4D scanning include: 3D scans to beused to generate automatic high-fidelity extreme poses and 4D scanswhich include high temporal resolution which enables mesh tracking toautomatically register extreme pose meshes for blending. Generatingautomatic high-fidelity extreme poses includes using 3D scans of theactor and muscle deformation of the actor to generate the automatichigh-fidelity extreme poses. 4D scanning and mesh-tracking are used toestablish topology correspondences across a 4D scanned mesh sequence forgenerating corrective shapes for shape interpolation and skeleton drivendeformation. The application is further configured for identifying andtargeting joints and muscles of the actor by the volumetric capturesystem for 3D scanning and 4D scanning Mesh generation includes muscleestimation or projection based on the 3D scanning and 4D scanning andmachine learning. Implementing mesh generation includes using the 3Dscanning and 4D scanning to generate meshes in extreme poses includingmuscle deformation. The application is further configured forimplementing mesh tracking for tracking topology changes to enable eachpose captured to have a same topology for blending between poses.

In another aspect, a system comprises a volumetric capture system for 3Dand 4D scanning including capturing photos and video simultaneously,wherein the 3D scanning and 4D scanning includes detecting muscledeformation of an actor and a computing device configured for: receivingthe captured photos and video from the volumetric capture system andimplementing mesh generation based on the 3D scanning and 4D scanning.The 3D scanning and 4D scanning include: 3D scans to be used to generateautomatic high-fidelity extreme poses and 4D scans which include hightemporal resolution which enables mesh tracking to automaticallyregister extreme pose meshes for blending. Generating automatichigh-fidelity extreme poses includes using 3D scans of the actor andmuscle deformation of the actor to generate the automatic high-fidelityextreme poses. 4D scanning and mesh-tracking are used to establishtopology correspondences across a 4D scanned mesh sequence forgenerating corrective shapes for shape interpolation and skeleton drivendeformation. The volumetric capture system is further configured foridentifying and targeting joints and muscles of the actor by thevolumetric capture system for 3D scanning and 4D scanning Meshgeneration includes muscle estimation or projection based on the 3Dscanning and 4D scanning and machine learning. Implementing meshgeneration includes using the 3D scanning and 4D scanning to generatemeshes in extreme poses including muscle deformation. The volumetriccapture system is further configured for implementing mesh tracking fortracking topology changes to enable each pose captured to have a sametopology for blending between poses.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a flowchart of a method of animating a subject usinga photo-video volumetric capture system according to some embodiments.

FIG. 2 illustrates a diagram of a mesh generated by combining a neutralpose and extreme poses according to some embodiments.

FIG. 3 illustrates a diagram of the correlation between human anatomyverus computer graphics according to some embodiments.

FIGS. 4A-B illustrate diagrams of muscle movements according to someembodiments.

FIG. 5 illustrates examples of major muscle groups according to someembodiments.

FIG. 6 illustrates a diagram of move types based on joints for meshcapture according to some embodiments.

FIG. 7 illustrates a diagram of move types based on joints for meshcapture according to some embodiments.

FIG. 8 illustrates examples of extreme poses according to someembodiments.

FIG. 9 illustrates a diagram of automatic blendshape extractionaccording to some embodiments.

FIG. 10 illustrates a flowchart of implementing mesh generationaccording to some embodiments.

FIG. 11 illustrates a block diagram of an exemplary computing deviceconfigured to implement the automatic blending method according to someembodiments.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

An automatic blending system utilizes an integrated photo-videovolumetric capture system for 3D/4D scanning acquires 3D scans and 4Dscans by acquiring images and videos simultaneously. The 3D scans areable to be used to generate auto high-fidelity extreme poses, and the 4Dscans include high temporal resolution which enables mesh tracking toautomatically register extreme pose meshes for blending.

A volumetric capture system (photo-video based) for high-quality 4Dscanning and mesh-tracking is able to be used to establish topologycorrespondences across a 4D scanned mesh sequence for generatingcorrective shapes which will be used in shape interpolation and skeletondriven deformation. The photo-video system aids mesh-tracking formaintaining mesh registration (topology consistency) along with ease ofextreme pose modeling unlike hand-crafted shape modeling which aidsregistration but has manual shape generation and the 3D scanning-basedapproach which aids shape generation but not registration.

The approach described herein is based on photo-video capture from a“photo-video volumetric capture system.” Photo-video based capture isdescribed in PCT Patent Application PCT/US2019/068151, filed Dec. 20,2019 titled, PHOTO-VIDEO BASED SPATIAL-TEMPORAL VOLUMETRIC CAPTURESYSTEM FOR DYNAMIC 4D HUMAN FACE AND BODY DIGITIZATION, which is herebyincorporated by reference in its entirety for all purposes. Asdescribed, the photo-video capture system is able to capture highfidelity texture in sparse time, and between the photo captures, videois captured, and the video is able to be used to establish thecorrespondence (e.g., transition) between the sparse photos. Thecorrespondence information is able to be used to implement meshtracking.

Major upper body and lower body joints are able to be identified thatare important for generating deformation and capturing the same using awide range of motion for all movement types across all joint categories.The joints are able to be used in muscle deformation. For example, byknowing how a joint moves and how a muscle near a joint deforms, theskeleton/joint information is able to be used for muscle deformationwhich is able to be used for mesh generation. Furthering the example,the images and videos acquired are also able to be used by having avideo of muscle deformation, the mesh of the muscle deformation is ableto be more accurately generated.

By using the photo-video system and mesh tracking, the topology changesare able to be tracked. Thus, each pose captured will have the sametopology which makes blending between multiple poses easier and moreaccurate.

FIG. 1 illustrates a flowchart of a method of animating a subject usinga photo-video volumetric capture system according to some embodiments.In the step 100, mesh creation/generation is implemented using theintegrated volumetric photo-video system. The mesh generation includesextreme pose modeling and registration for blending. As described, theintegrated photo-video volumetric capture system for 3D/4D scan acquires3D scans and 4D scans by acquiring images and videos of a subject/actorsimultaneously. The 3D scans are able to be used to generate autohigh-fidelity extreme poses, and the 4D scans include high temporalresolution which enables mesh tracking to automatically register extremepose meshes for blending. In the step 102, skeleton fitting isimplemented. Skeleton fitting is able to be implemented in any mannersuch as based on relative marker trajectories. In the step 104, skinweight painting is performed. Skin weight painting is able to beimplemented in any manner such as determining the weight of each segmentof skin and painting accordingly. In the step 104, animation isperformed. Animation is able to be performed in any manner. Depending onthe implementation, each of the steps is able to be performed manually,semi-automatically or automatically. In some embodiments, fewer oradditional steps are implemented. In some embodiments, the order of thesteps is modified.

FIG. 2 illustrates a diagram of a mesh generated by combining a neutralpose and extreme poses according to some embodiments. A neutral pose isable to be any standard pose such as standing with arms down, arms up orarms out to the side. Extreme poses are the poses between standard posessuch as when a subject moves between standard poses. Extreme poses arecaptured by targeting specific parts of the human muscle, which enablesgeneration of the extreme shape for the game development pipeline. Thephoto-video system and mesh tracking are able to be used to target allmuscle groups of the human body to capture and solve the problem ofmaintaining a mesh registration in the graphics game developmentpipeline.

When developing a new video game, a model is captured for the game. Anactor typically comes in to a studio one time to be recorded performingspecified movements and/or actions. The studio comprehensively capturesall of the actor's muscle deformations using the photo-video volumetriccapture system. Moreover, by using existing kinesiology movements andtypes of deformation that occur in the human body, a corresponding meshis able to have similar deformations. Using a previously capturedneutral poses and additional captured poses, a system is able to deformthe model to be similar to human movements/deformations. Additionally,the kinesiology movements, deformations and/or other knowledge and dataare able to be used in training the system.

FIG. 3 illustrates a diagram of the correlation between human anatomyverus computer graphics according to some embodiments. In human anatomy,musculoskeletal actuation involves receiving a signal from a person'smotor cortex. Then, muscle deformation occurs which enables joint/bonemovement by the muscle pulling on the bone. Additionally, there isskin/fat movement. In a computer graphics mesh, a motion driver triggersmovement in an animated character, specifically by performing joint/bonemovement. Mesh deformation (Skeletal Subspace Deformation (SSD)) thenoccurs, followed by mesh deformation (Pose Space Deformation (PSD)). Aclear correlation is able to be seen between human anatomy and a meshgenerated using computer graphics.

FIGS. 4A-B illustrate diagrams of muscle movements according to someembodiments. Human body parts bend at joints as shown such as the headbending at the neck, hands bending at the wrist, fingers bending atknuckles, legs bending at the knee, and feet bending at the ankle Insome embodiments, all joint movements are able to be fit into 12categories. In some embodiments, by classifying the joint movements intocategories, the correct muscle deformation is able to be generated basedon the classified movement. For example, when a character bends at theknee, specific muscles deform in the leg, and using machine learning,the correct muscles are able to be deformed at the appropriate time. Themuscle movements are the types of movements the actor will performincluding the range of motion. The muscle movements are targeted forcapture. [FIGS. 4A-B, DeSaix, Peter, et al. “Anatomy & Physiology(OpenStax).” (2013). (Retrieved fromhttps://openlibrary-repo.ecampusontario.ca/jspui/handle/123456789/331)]

FIG. 5 illustrates examples of major muscle groups according to someembodiments. The upper body and the lower body each have 4 joints(excluding finger/toe joints). The joints in the upper body include: theshoulder, elbow, neck and hands, and the joints in the lower bodyinclude: the torso, hips, knees and ankles Each of the joints havecorresponding muscle groups. As described, these corresponding musclegroups deform when the character is in motion. The lower body and upperbody muscles are the main targets for capture when an actor is moving.

FIG. 6 illustrates a diagram of move types based on joints for meshcapture according to some embodiments. There are many different movetypes with varying angular ranges of motion (0 to 180 degrees) for eachof the main upper and lower joints. By including the various move types,the desired muscles are able to be captured and then later utilized whengenerating the mesh.

FIG. 7 illustrates a diagram of move types based on joints for meshcapture according to some embodiments. Two of the 12 move types areshown (flexion/extension and pronation/supination). In some embodiments,the angular range of motion is selectable from 0, 90 and 180 degrees,and in some embodiments, finer tuning of the angular range of motion ispossible to a specific number of degrees or even fractions of degrees.

FIG. 8 illustrates examples of extreme poses according to someembodiments. Image 800 shows six movement types such as lifting arms upto the side, raising arms from down at the hips to over head and puttingarms out front. Image 802 shows the four joints and the target muscles.

FIG. 9 illustrates a diagram of automatic blendshape extractionaccording to some embodiments. Pose parameters 900 combined with facialaction units 902 result in the 4D tracked meshes 904. An automaticblendshape extraction method uses 4D scans of a moving face whichexpedites the character making process and reduces the production cost.A 4D face scan method such as U.S. patent application Ser. No.17/411,432, filed Aug. 25, 2021, titled, “PRESERVING GEOMETRY DETAILS INA SEQUENCE OF TRACKED MESHES,” which is hereby incorporated by referencein its entirety for all purposes, is able to be used. It provides highquality 4D tracked meshes of the moving face as shown in 904, and thepose parameters 900 also are able to be obtained from the tracked 4Dmeshes. The user may use control points or bones for the poserepresentation. [FIG. 9, center figure is from P. Ekman, Wallace V.Friesen, Joseph C. Hager, “Facial action coding system: A technique forthe measurement of facial movement>>Psychology 1978, 2002. ISBN0-931835-01-1 1.]

Facial action units are of interest. With 4D tracked meshes including avariety of different expressions available, a set of character-specificfacial action units are able to be automatically generated. It can beregarded as a decomposition of 4D meshes into dynamic pose parametersand static action units, where only the action units are unknown.Machine learning techniques for the decomposition problem are able to beused.

FIG. 10 illustrates a flowchart of implementing mesh generationaccording to some embodiments. In the step 1000, a volumetric capturesystem is utilized for high-quality 3D/4D scanning. As described in PCTPatent Application PCT/US2019/068151, the volumetric capture system isable to acquire photos and videos simultaneously for high-quality 3D/4Dscanning. The high-quality 3D/4D scanning includes denser camera viewsfor high-quality modeling. In some embodiments, instead of utilizing thevolumetric capture system, another system for acquiring 3D content andtime information is utilized. For example, at least two separate 3Dscans are acquired. Furthering the example, the separate 3D scans areable to be captured and/or downloaded.

During the capture time, joint and muscle movement and deformation areacquired. For example, the specific muscles and the specific deformationof the muscles over time are captured. Specific joints and thecorresponding muscles to the joints of the actor are able to be targetedduring capture time. For example, the target subject/actor is able to berequested to move, and the muscles will deform. The deformation of themuscles is able to be captured statically and in motion. The informationacquired from the movement and deformation is able to be used to train asystem so that the system is able to use the joint and muscleinformation to perform any movement of the character. For a very complexsituation, this is very difficult for an animator to do. Any complexmuscle deformation is learned during the modeling stage. This enablessynthesis in the animation stage.

In the step 1002, mesh generation is implemented. Once high qualityinformation is captured for the scanning, mesh generation is implementedincluding extreme pose modeling and registration for blending. The 3Dscan information is able to be used to generate auto high-fidelityextreme poses. For example, the frames between key frames are able to beproperly generated using the 4D scan information which includes frameinformation between key frames. The high temporal resolution of the 4Dscan information enables mesh tracking to auto register extreme posemeshes for blending. In another example, the 4D scan enables meshgeneration of a muscle deforming over time. Similarly, with machinelearning involving joint information as well as corresponding muscle andmuscle deformation information, a mesh including muscle deformationinformation is able to be generated where the movement was not acquiredby the capture system. For example, although an actor was requested toperform a standing vertical jump and to run for capture, the capturesystem did not acquire the actor performing a running jump. However,based on the acquired information of the standing vertical jump andrunning where the acquired information includes muscle deformationduring those actions, and using the machine learning with the knowledgeof joints and other physiological information, the mesh for the runningjump including detailed muscle deformation is able to be generated. Insome embodiments, mesh generation includes muscle estimation orprojection based on the 3D scanning and 4D scanning and machinelearning.

Major upper body and lower body joints are able to be identified thatare important for generating deformation and capturing deformation usinga wide range of motion for all movement types across all jointcategories.

By using the volumetric capture system and mesh tracking, the topologychanges are able to be tracked. Thus, each pose captured will have thesame topology which makes blending between multiple poses easier andmore accurate. The targeted joints and muscles are able to be utilizedwhen generating the mesh.

In some embodiments, mesh generation includes generating a static meshbased on the 3D scan information, and the mesh is able to bemodified/animated using the 4D scan information. For example, as themesh moves in time, additional mesh information is able to beestablished/generated from the video content of the 4D scan informationand/or machine learning information. As described, the transitionsbetween each frame of the animated mesh are able to maintain topology,such that the mesh tracking and blending is smooth. In other words,topology correspondences are established across a 4D scanned meshsequence for generating corrective shapes which will be used in shapeinterpolation and skeleton driven deformation.

In some embodiments, fewer or additional steps are implemented. In someembodiments, the order of the steps is modified.

FIG. 11 illustrates a block diagram of an exemplary computing deviceconfigured to implement the automatic blending method according to someembodiments. The computing device 1100 is able to be used to acquire,store, compute, process, communicate and/or display information such asimages and videos. The computing device 1100 is able to implement any ofthe automatic blending aspects. In general, a hardware structuresuitable for implementing the computing device 1100 includes a networkinterface 1102, a memory 1104, a processor 1106, I/O device(s) 1108, abus 1110 and a storage device 1112. The choice of processor is notcritical as long as a suitable processor with sufficient speed ischosen. The memory 1104 is able to be any conventional computer memoryknown in the art. The storage device 1112 is able to include a harddrive, CDROM, CDRW, DVD, DVDRW, High Definition disc/drive, ultra-HDdrive, flash memory card or any other storage device. The computingdevice 1100 is able to include one or more network interfaces 1102. Anexample of a network interface includes a network card connected to anEthernet or other type of LAN. The I/O device(s) 1108 are able toinclude one or more of the following: keyboard, mouse, monitor, screen,printer, modem, touchscreen, button interface and other devices.Automatic blending application(s) 1130 used to implement the automaticblending method are likely to be stored in the storage device 1112 andmemory 1104 and processed as applications are typically processed. Moreor fewer components shown in FIG. 11 are able to be included in thecomputing device 1100. In some embodiments, automatic blending hardware1120 is included. Although the computing device 1100 in FIG. 11 includesapplications 1130 and hardware 1120 for the automatic blending method,the automatic blending method is able to be implemented on a computingdevice in hardware, firmware, software or any combination thereof. Forexample, in some embodiments, the automatic blending applications 1130are programmed in a memory and executed using a processor. In anotherexample, in some embodiments, the automatic blending hardware 1120 isprogrammed hardware logic including gates specifically designed toimplement the automatic blending method.

In some embodiments, the automatic blending application(s) 1130 includeseveral applications and/or modules. In some embodiments, modulesinclude one or more sub-modules as well. In some embodiments, fewer oradditional modules are able to be included.

Examples of suitable computing devices include a personal computer, alaptop computer, a computer workstation, a server, a mainframe computer,a handheld computer, a personal digital assistant, a cellular/mobiletelephone, a smart appliance, a gaming console, a digital camera, adigital camcorder, a camera phone, a smart phone, a portable musicplayer, a tablet computer, a mobile device, a video player, a video discwriter/player (e.g., DVD writer/player, high definition discwriter/player, ultra high definition disc writer/player), a television,a home entertainment system, an augmented reality device, a virtualreality device, smart jewelry (e.g., smart watch), a vehicle (e.g., aself-driving vehicle) or any other suitable computing device.

To utilize the automatic blending method described herein, devices suchas digital cameras/camcorders/computers are used to acquire content andthen the same devices or one or more additional devices analyze thecontent. The automatic blending method is able to be implemented withuser assistance or automatically without user involvement to performautomatic blending.

In operation, the automatic blending method provides a more accurate andefficient automatic blending and animation method. The automaticblending method utilizes a photo-video system which aids mesh-trackingfor maintaining mesh registration (topology consistency) along with easeof extreme pose modeling unlike hand-crafted shape modeling which aidsregistration but has manual shape generation and the 3D scanning-basedapproach which aids shape generation but not registration. By using thephoto-video system and mesh tracking, the topology changes are able tobe tracked. Thus, each pose captured will have the same topology whichmakes blending between multiple poses easier and more accurate.

Some Embodiments of Automatic Blending of Human Facial Expression andFull-Body Poses for Dynamic Digital Human Model Creation UsingIntegrated Photo-Video Volumetric Capture System and Mesh-Tracking

-   1. A method programmed in a non-transitory of a device comprising:

using a volumetric capture system configured for 3D scanning and 4Dscanning including capturing photos and video simultaneously, whereinthe 3D scanning and 4D scanning includes detecting muscle deformation ofan actor; and

implementing mesh generation based on the 3D scanning and 4D scanning

-   2. The method of clause 1 wherein the 3D scanning and 4D scanning    include:

3D scans to be used to generate automatic high-fidelity extreme posesand

4D scans which include high temporal resolution which enables meshtracking to automatically register extreme pose meshes for blending.

-   3. The method of clause 2 wherein generating automatic high-fidelity    extreme poses includes using 3D scans of the actor and muscle    deformation of the actor to generate the automatic high-fidelity    extreme poses.-   4. The method of clause 2 wherein 4D scanning and mesh-tracking are    used to establish topology correspondences across a 4D scanned mesh    sequence for generating corrective shapes for shape interpolation    and skeleton driven deformation.-   5. The method of clause 1 further comprising identifying and    targeting joints and muscles of the actor by the volumetric capture    system for 3D scanning and 4D scanning-   6. The method of clause 1 wherein mesh generation includes muscle    estimation or projection based on the 3D scanning and 4D scanning    and machine learning.-   7. The method of clause 1 wherein implementing mesh generation    includes using the 3D scanning and 4D scanning to generate meshes in    extreme poses including muscle deformation.-   8. The method of clause 1 further comprising implementing mesh    tracking for tracking topology changes to enable each pose captured    to have a same topology for blending between poses.-   9. An apparatus comprising:

a non-transitory memory for storing an application, the application for:

-   -   using a volumetric capture system configured for 3D scanning and        4D scanning including capturing photos and video simultaneously,        wherein the 3D scanning and 4D scanning includes detecting        muscle deformation of an actor; and    -   implementing mesh generation based on the 3D scanning and 4D        scanning; and

a processor coupled to the memory, the processor configured forprocessing the application.

-   10. The apparatus of clause 9 wherein the 3D scanning and 4D    scanning include:

3D scans to be used to generate automatic high-fidelity extreme posesand

4D scans which include high temporal resolution which enables meshtracking to automatically register extreme pose meshes for blending.

-   11. The apparatus of clause 10 wherein generating automatic    high-fidelity extreme poses includes using 3D scans of the actor and    muscle deformation of the actor to generate the automatic    high-fidelity extreme poses.-   12. The apparatus of clause 10 wherein 4D scanning and mesh-tracking    are used to establish topology correspondences across a 4D scanned    mesh sequence for generating corrective shapes for shape    interpolation and skeleton driven deformation.-   13. The apparatus of clause 9 wherein the application is further    configured for identifying and targeting joints and muscles of the    actor by the volumetric capture system for 3D scanning and 4D    scanning-   14. The apparatus of clause 9 wherein mesh generation includes    muscle estimation or projection based on the 3D scanning and 4D    scanning and machine learning.-   15. The apparatus of clause 9 wherein implementing mesh generation    includes using the 3D scanning and 4D scanning to generate meshes in    extreme poses including muscle deformation.-   16. The apparatus of clause 9 wherein the application is further    configured for implementing mesh tracking for tracking topology    changes to enable each pose captured to have a same topology for    blending between poses.-   17. A system comprising:

a volumetric capture system for 3D and 4D scanning including capturingphotos and video simultaneously, wherein the 3D scanning and 4D scanningincludes detecting muscle deformation of an actor; and

a computing device configured for:

-   -   receiving the captured photos and video from the volumetric        capture system; and    -   implementing mesh generation based on the 3D scanning and 4D        scanning

-   18. The system of clause 17 wherein the 3D scanning and 4D scanning    include:

3D scans to be used to generate automatic high-fidelity extreme posesand

4D scans which include high temporal resolution which enables meshtracking to automatically register extreme pose meshes for blending.

-   19. The system of clause 18 wherein generating automatic    high-fidelity extreme poses includes using 3D scans of the actor and    muscle deformation of the actor to generate the automatic    high-fidelity extreme poses.-   20. The system of clause 18 wherein 4D scanning and mesh-tracking    are used to establish topology correspondences across a 4D scanned    mesh sequence for generating corrective shapes for shape    interpolation and skeleton driven deformation.-   21. The system of clause 17 wherein the volumetric capture system is    further configured for identifying and targeting joints and muscles    of the actor by the volumetric capture system for 3D scanning and 4D    scanning-   22. The system of clause 17 wherein mesh generation includes muscle    estimation or projection based on the 3D scanning and 4D scanning    and machine learning.-   23. The system of clause 17 wherein implementing mesh generation    includes using the 3D scanning and 4D scanning to generate meshes in    extreme poses including muscle deformation.-   24. The system of clause 17 wherein the volumetric capture system is    further configured for implementing mesh tracking for tracking    topology changes to enable each pose captured to have a same    topology for blending between poses.

The present invention has been described in terms of specificembodiments incorporating details to facilitate the understanding ofprinciples of construction and operation of the invention. Suchreference herein to specific embodiments and details thereof is notintended to limit the scope of the claims appended hereto. It will bereadily apparent to one skilled in the art that other variousmodifications may be made in the embodiment chosen for illustrationwithout departing from the spirit and scope of the invention as definedby the claims.

What is claimed is:
 1. A method programmed in a non-transitory of adevice comprising: using a volumetric capture system configured for 3Dscanning and 4D scanning including capturing photos and videosimultaneously, wherein the 3D scanning and 4D scanning includesdetecting muscle deformation of an actor; and implementing meshgeneration based on the 3D scanning and 4D scanning.
 2. The method ofclaim 1 wherein the 3D scanning and 4D scanning include: 3D scans to beused to generate automatic high-fidelity extreme poses and 4D scanswhich include high temporal resolution which enables mesh tracking toautomatically register extreme pose meshes for blending.
 3. The methodof claim 2 wherein generating automatic high-fidelity extreme posesincludes using 3D scans of the actor and muscle deformation of the actorto generate the automatic high-fidelity extreme poses.
 4. The method ofclaim 2 wherein 4D scanning and mesh-tracking are used to establishtopology correspondences across a 4D scanned mesh sequence forgenerating corrective shapes for shape interpolation and skeleton drivendeformation.
 5. The method of claim 1 further comprising identifying andtargeting joints and muscles of the actor by the volumetric capturesystem for 3D scanning and 4D scanning.
 6. The method of claim 1 whereinmesh generation includes muscle estimation or projection based on the 3Dscanning and 4D scanning and machine learning.
 7. The method of claim 1wherein implementing mesh generation includes using the 3D scanning and4D scanning to generate meshes in extreme poses including muscledeformation.
 8. The method of claim 1 further comprising implementingmesh tracking for tracking topology changes to enable each pose capturedto have a same topology for blending between poses.
 9. An apparatuscomprising: a non-transitory memory for storing an application, theapplication for: using a volumetric capture system configured for 3Dscanning and 4D scanning including capturing photos and videosimultaneously, wherein the 3D scanning and 4D scanning includesdetecting muscle deformation of an actor; and implementing meshgeneration based on the 3D scanning and 4D scanning; and a processorcoupled to the memory, the processor configured for processing theapplication.
 10. The apparatus of claim 9 wherein the 3D scanning and 4Dscanning include: 3D scans to be used to generate automatichigh-fidelity extreme poses and 4D scans which include high temporalresolution which enables mesh tracking to automatically register extremepose meshes for blending.
 11. The apparatus of claim 10 whereingenerating automatic high-fidelity extreme poses includes using 3D scansof the actor and muscle deformation of the actor to generate theautomatic high-fidelity extreme poses.
 12. The apparatus of claim 10wherein 4D scanning and mesh-tracking are used to establish topologycorrespondences across a 4D scanned mesh sequence for generatingcorrective shapes for shape interpolation and skeleton drivendeformation.
 13. The apparatus of claim 9 wherein the application isfurther configured for identifying and targeting joints and muscles ofthe actor by the volumetric capture system for 3D scanning and 4Dscanning.
 14. The apparatus of claim 9 wherein mesh generation includesmuscle estimation or projection based on the 3D scanning and 4D scanningand machine learning.
 15. The apparatus of claim 9 wherein implementingmesh generation includes using the 3D scanning and 4D scanning togenerate meshes in extreme poses including muscle deformation.
 16. Theapparatus of claim 9 wherein the application is further configured forimplementing mesh tracking for tracking topology changes to enable eachpose captured to have a same topology for blending between poses.
 17. Asystem comprising: a volumetric capture system for 3D and 4D scanningincluding capturing photos and video simultaneously, wherein the 3Dscanning and 4D scanning includes detecting muscle deformation of anactor; and a computing device configured for: receiving the capturedphotos and video from the volumetric capture system; and implementingmesh generation based on the 3D scanning and 4D scanning.
 18. The systemof claim 17 wherein the 3D scanning and 4D scanning include: 3D scans tobe used to generate automatic high-fidelity extreme poses and 4D scanswhich include high temporal resolution which enables mesh tracking toautomatically register extreme pose meshes for blending.
 19. The systemof claim 18 wherein generating automatic high-fidelity extreme posesincludes using 3D scans of the actor and muscle deformation of the actorto generate the automatic high-fidelity extreme poses.
 20. The system ofclaim 18 wherein 4D scanning and mesh-tracking are used to establishtopology correspondences across a 4D scanned mesh sequence forgenerating corrective shapes for shape interpolation and skeleton drivendeformation.
 21. The system of claim 17 wherein the volumetric capturesystem is further configured for identifying and targeting joints andmuscles of the actor by the volumetric capture system for 3D scanningand 4D scanning.
 22. The system of claim 17 wherein mesh generationincludes muscle estimation or projection based on the 3D scanning and 4Dscanning and machine learning.
 23. The system of claim 17 whereinimplementing mesh generation includes using the 3D scanning and 4Dscanning to generate meshes in extreme poses including muscledeformation.
 24. The system of claim 17 wherein the volumetric capturesystem is further configured for implementing mesh tracking for trackingtopology changes to enable each pose captured to have a same topologyfor blending between poses.